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Program description 

Positive Action, a K-12 program, aims to promote a positive 
self-concept, character development, healthy behavior, and 
social-emotional skills and to reduce disruptive and problem 
behavior. The curricula for elementary and middle school 
students each have seven units per kit, with nine kits per grade 
level. The high school curriculum has six units per kit, with four 
kits per grade level. But the overall number of lessons may 

vary by grade level. All lessons are scripted and use classroom 
discussion, role-playing, games, songs, and activity sheets or 
text booklets. Optional components that may or may not be 
implemented as part of the program are: climate development; 
drug education for grade 5 and middle school; conflict resolu- 
tion plan; community, parent, and family classes; and counselor 
components. 


Research One study of Positive Action met the What Works Clearinghouse Florida and Hawaii. Both studies examined results on students’ 
(WWC) evidence standards, and one study met standards with behavior and academic achievement, 
reservations. The studies included 56 elementary schools in 


Effectiveness Positive Action was found to 


Rating of effectiveness 

Improvement index 1 2 Average: +19 percentile points Not reported Average: +15 percentile points 

Range: -12 to +36 percentile points Range: +8 to +36 percentile points 


have potentially positive effects on elementary school students’ behavior and academic achievement. 

Knowledge, attitudes, 

Behavior and values Academic achievement 

Positive effects Not reported Positive effects 


1. The evidence presented in this report is based on available research. Findings and conclusions may change as new research becomes available. 

2. These numbers show the average and range of improvement indices for all findings across the two studies. 
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Developer and contact 

Positive Action, Inc. 264 4th Avenue South, Twin Falls, ID 83301. 
Web: http://www.positiveaction.net . E-mail: info@positiveaction. 
net. Telephone: (800) 345-2974. Fax: (208) 733-1590. 

Scope of use 

The Positive Action program was created and first implemented 
in four school sites in Idaho from 1979-81. Positive Action, Inc. 
was founded in 1982. Since then, more than 11,000 schools in 
2,500 school districts and 2,000 community groups and agencies 
in all 50 states have adopted the program. Positive Action may 
have changed since the studies were conducted. The WWC 
recommends asking the developer for information about the most 
current version of this curriculum and taking into account that 
student demographics and school context may affect outcomes. 

Teaching 

The Positive Action curriculum for each of the elementary school 
grades averages 140 lessons taught four days a week, organized 
into six teaching units and a seventh review unit, presented in 
a consistent order. Units 1-3 and a summary of units 4-7 are 
taught in grade 7, totaling 81 lessons, while units 4-7 and a 
summary of units 1-3 are taught in grade 8, totaling 75 lessons. 
The high school curriculum is composed of four kits. Each kit 
covers one school year but can stand on its own or be used in a 
different order. Each kit, with six teaching units and a review unit, 
has approximately 132 lessons. Kit IV, however, has two options, 
each with its own curriculum, so the possible number of lessons 
doubles for that kit. In each of the elementary, middle, or high 
school grade levels teachers may implement the entire set of 
lessons or choose a partial implementation of at least 20 lessons 
per grade level. There are Levels of Implementation Plans for the 
Bronze, Silver, Gold, and Platinum Levels, which guide users on 


how to choose lessons for the highest level of effectiveness. In 
all grade levels a typical lesson lasts about 15 minutes. 

The Positive Action program consists of a core curriculum 
and optional components that may or may not be implemented 
as part of the program — sitewide climate development; drug 
education for grade 5 and middle school; conflict resolution; 
counselor, parent, and family classes; and community/coalition 
development. The program philosophy, “You feel good about 
yourself when you think and do positive actions, and there is 
a positive way to do everything,” underlies the content of the 
program units. Kits for the core curriculum and the supplemental 
components may include instructor’s manuals, group materials, 
activity booklets or activity sheets, text booklets, posters, song 
CDs with lyrics, game boards, and teaching aides (stickers, 
index cards, medals, puppets, flannel boards and figures, bean 
bags, and visual aids). 

Cost 

The cost of a Positive Action classroom kit varies by grade level. 
Kindergarten kits cost $460; grades 1-8 cost $360; high school 
kits I, II, and III cost $360; and high school kit IV costs $460. 

The cost of the drug education supplemental kits also varies by 
grade level, ranging from $250 to $360. The optional community 
kit costs $550, the counselor kit $125, and the family kit $75. 
Family classes cost $360 and parenting classes $160. Some of 
the kits are available in Spanish. Combo kits (at reduced prices), 
refresher kits, and kit parts are also available for varied prices. 

An orientation training workshop that includes curriculum; 
climate development; and counselor, family, and community 
components costs $200 a school type (elementary, middle, or 
high school). Ongoing training workshops and media training 
workshops cost $250 each. Workshops that combine orientation, 
ongoing training, and media training for grades K-12 cost $900. 


Thirteen studies reviewed by the WWC investigated the effects 
of the Positive Action program. One study of the impact of Posi- 
tive Action on elementary school students met WWC evidence 


standards, and one study met standards with reservations. One 
study (Flay, Acock, Vuchinich, & Beets, 2006) was a randomized 
controlled trial. The second study (Flay & Allred, 2003) used a 
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Research (continued) 


Effectiveness 


The WWC found Positive 
Action to have positive 
effects on both behavior and 
academic achievement of 
elementary school students 
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quasi-experimental design. The remaining 10 studies did not 
meet WWC evidence screens. 

Flay and Allred (2003) included 36 elementary schools in 
a large Southeastern school district. The study compared 


outcomes for students participating in the Positive Action 
program with the outcomes for students in schools that did 
not use this program. The study focused on Positive Action as 
implemented schoolwide. 


Findings 

The WWC review of character education addresses student 
outcomes in three domains: behavior; knowledge, attitudes, and 
values; and academic achievement. 

Behavior. Flay and colleagues (2006) reported statistically sig- 
nificant differences favoring the intervention groups on students’ 
suspension rates, use of alcohol, being drunk, and use of tobacco 
and illegal drugs. The study also reported statistically significant 
differences favoring the intervention group on serious violence 
among boys but not among girls. The average effect size across 
all behavior outcomes in this study was statistically significant. 3 

Flay and Allred (2003) reported statistically significant dif- 
ferences favoring the intervention group on students’ violent 
behavior and suspension rates. In addition, the average effect 
size was statistically significant. 

Academic achievement. Flay and colleagues (2006) reported, 
and the WWC confirmed, statistically significant differences 
favoring the intervention groups on students’ grade retention 
rates. The study also examined the impact on state standard- 


ized test scores in reading and math but found no statistically 
significant differences. In addition, the average effect size was 
statistically significant. 

Flay and Allred (2003) reported, and the WWC confirmed, 
statistically significant differences favoring the intervention 
group on total scores of the Florida Comprehensive Aptitude 
Test (FCAT). The authors reported no statistically significant 
impacts on absenteeism. The average effect size across the two 
outcomes in this domain was statistically significant. 

Rating of effectiveness 

The WWC rates interventions as positive, potentially positive, 
mixed, no discernible effects, potentially negative, or negative. 
The rating of effectiveness takes into account four factors: the 
quality of the research design, the statistical significance of the 
findings (as calculated by the WWC), the size of the difference 
between participants in the intervention condition and the com- 
parison condition, and the consistency in findings across studies 
(see the WWC Intervention Rating Scheme) . 


Improvement index 

For each outcome domain, the WWC computed an improvement 
index based on the effect size (see the Technical Details of 
WWC-Conducted Computations) . The improvement index repre- 
sents the difference between the percentile rank of the average 
student in the intervention condition versus the percentile rank 


of the average student in the comparison condition. Unlike the 
rating of effectiveness, the improvement index is entirely based 
on the size of the effect, regardless of the statistical significance 
of the effect, the study design, or the analysis. The improvement 
index can take on values between -50 and +50, with positive 
numbers denoting favorable results. The improvement index 


3. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and 
for multiple comparisons. For an explanation, see the WWC Tutorial on Mismatch . See the Technical Details of WWC-Conducted Computations for the formulas the WWC 
used to calculate the statistical significance. In the case of Positive Action, corrections for both clustering and multiple comparisons were needed. 
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References 


for behavior is +19 percentile points, with a range of -12 to 
+36 percentile points. The improvement index for academic 
achievement is +15 percentile points, with a range of +8 to +36 
percentile points. 

Summary 

The WWC reviewed 13 studies on Positive Action. One study met 
WWC evidence standards, and one study met standards with 


reservations. Both studies assessed elementary school student 
outcomes in the behavior and academic achievement domains. 
When the WWC aggregated the results in each of these domains, 
the average effect sizes were statistically significant. So the WWC 
rated the program as having positive effects on both behavior and 
academic achievement. Character education, an evolving field, is 
beginning to establish a research base. The evidence presented in 
this report is limited and may change as new research emerges. 


Met WWC evidence standards 

Flay, B., Acock, A., Vuchinich, S., & Beets, M. (2006). Progress 
report of the randomized trial of Positive Action in Hawaii: End 
of third year of intervention. Available from Positive Action, 

Inc. 264 4th Avenue South, Twin Falls, ID 83301. 

Met WWC evidence standards with reservations 

Flay, B. R., & Allred, C. G. (2003). Long-term effects of the Posi- 
tive Action program. American Journal of Healthy Behavior, 
27(1), 6-21. 

Did not meet WWC evidence screens 

Allred, C. G. (1984). The Positive Action program: An evaluation. 
Royal School, Honolulu School District, HI. Twin Falls, ID: 
Positive Action, Inc. 4 

Burcham, S. (1992). Improving the academic self-esteem of 
elementary students. Unpublished doctoral dissertation, Nova 
University, Fort Lauderdale-Davie, FL. 5 6 

Flay, B. R., & Ordway, N. (1999). Effectiveness of the Positive Action 
program: Changes in school-level percentile achievement rank- 
ings and disciplinary report. Twin Falls, ID: Positive Action, Inc. 5 


Flay, B. R. (2001). An intensive case study of the Positive Action 
program as a comprehensive school reform demonstration 
program: Year 2 results. Twin Falls, ID: Positive Action, Inc. 5 

Additional citation 

Flay, B. R. (2000). An intensive case study of the Positive 
Action® program as a comprehensive school reform dem- 
onstration program. Twin Falls, ID: Positive Action, Inc. 

Flay, B. R., Allred, C. G., & Ordaway, N. (2001). Effects of the 
Positive Action program on achievement and discipline: Two 
matched control comparisons. Prevention Science, 2(2), 
71-88.® 

Flay, B. R., & Slagel, M. (2006, July). The Positive Action Family 
Program: A pilot randomized trial. Available from the Positive 
Action, Inc., 264 4th Avenue South, Twin Falls, ID 83301. 7 
Heuer, L. G. (1995). Behavior, attitudes, and knowledge related 
to drug and alcohol prevention curricula in North Dakota sev- 
enth through twelve grade students. Dissertation Abstracts 
International, 56(10), 3826. (UMI No. 9605472). 5 
Hoyer-Rufner, C. S. (1994). The Positive Action curriculum for 
fifth graders: An exploratory study of accuracy of self-percep- 
tions and self-esteem. Dissertation Abstracts International, 
54(7), 2513A. (UMI No. 9333361). 5 


4. Does not use a strong causal design: there was only one intervention and one comparison unit, so the analysis could not separate the effects of the 
intervention from other factors. 

5. Does not use a strong causal design: the study did not use a comparison group. 

6. Does not use a strong causal design: the study, which used a quasi-experimental design, did not establish that the comparison group was equivalent to 
the treatment group at the baseline 

7. The sample is not appropriate to this review. 
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self-esteem of 6th-grade students. Dissertation Abstracts 
International, 52 ( 07), 2417. (UMI No. 9136806). 4 
West, B. G. (1997). The effects of improving the self-esteem on 
academic performance of fourth-grade students in a rural 


elementary school. Dissertation Abstracts International, 
58(03), 691. (UMI No. 9726368). 8 

Woodward, J. R. (1996). Improving academic achievement of 
fourth-grade students through a program of self-concept 
enhancement activities. Unpublished doctoral dissertation, 
Nova Southeastern University, Jacksonville, FL. 8 


For more information about specific studies and WWC calculations, please see the WWC Positive Action 
Technical Appendices . 


8. Complete data are not reported: the WWC could not compute effect sizes. 
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Appendix 


Appendix A1.1 Study characteristics: Flay, Acock, Vuchinich, & Beets, 2006 (randomized controlled trial) 


Characteristic 

Description 

Study citation 

Flay, B., Acock, A., Vuchinich, S., & Beets, M. (2006). Progress report of the randomized trial of Positive Action in Hawaii: End of third year of intervention. Available from 
Positive Action, Inc. 264 4th Avenue South, Twin Falls, ID 83301. 

Participants 

The study tracked students who started grades K-1 in the 2001-02 academic year. The report reviewed by the WWC examined the outcomes of 2,666 third- and fourth- 
grade students at the end of the third year of program implementation. These outcomes included students who had remained in the school since the beginning of the study 
as well as new students in the school. About 25% of the students surveyed each year were new to the school. The students were enrolled in 20 elementary schools that were 
randomly assigned into conditions. A small percentage of the students were white (13.40% in the intervention school and 17.94% in the comparison). Among the remaining 
students the following ethnic groups were represented: Hawaiian, Filipino, Asian (other), Japanese, Samoan, Hispanic, Chinese, Black, and Portuguese. About 60% of the 
schools in the sample were Title 1 schools. 

Setting 

The study was conducted in 20 K-5 or K-6 schools in Hawaii. Five pairs of matched participating schools were from Oahu, three pairs were from Maui, and two pairs were 
from Molokai. 

Intervention 

The program consisted of the Positive Action curriculum and additional components that involved school principals, counselors, parents, and community members. The 
components addressed school and classroom management, school climate, family and community involvement, and skills and knowledge related to core values. 1 The report 
reviewed by the WWC presented findings for the end of the third year of program implementation. The authors report that by the third year, two schools were still implementing 
at a very low level, three at a moderate-to-high level, and five at a high level. But even the high-implementation schools were still not implementing at the level the program 
developer would expect for high-implementation schools. For example, few schools implemented the family- or community-involvement programs. 

Comparison 

The comparison schools were drawn from the same school districts as the intervention schools and were matched on demographic characteristics, student behavior, and 
academic achievement. Comparison schools did not implement the Positive Action program. The comparison schools had other types of character education activities they 
regarded as business-as-usual, which were also practiced (although to a lesser extent) in the intervention schools. 1 

Primary outcomes 
and measurement 

The study examined students’ outcomes in the academic achievement and behavior domains. 2 Outcomes in the academic achievement domain included percent proficient on 
the reading and math sections of a state standardized test and daily absences. Outcomes in the behavior domain included students’ reports of use of alcohol, tobacco, and 
illegal drugs. (See Appendices A2.1 and A2.2 for more detailed descriptions of outcome measures.) 

Teacher training 

Prior to each school year, the program developer, Dr. Allred, provided teacher/staff training to each intervention school. The training lasted 3-4 hours the first year and 1-2 
hours each of the subsequent years. In addition, Dr. Allred visited each school at least once each year to provide an in-service session (usually 30-50 minutes). Finally, 5-6 
representatives from each intervention school participated each winter in a mini-conference to obtain further training on the schoolwide components of the program and to 
share experiences. 


1 . Schools in the intervention and comparison groups implemented additional character and behavior programs at the time of the study, which they regarded as business-as-usual. These programs include the Hawai’i State 
Department of Education (HIDOE) mandated General Learner Outcomes (GLO), TRIBES, Lions Quest, and additional programs related to school management and organization and drug and violence prevention, such 

as Positive Behavioral Support, D.A.R.E., Red Ribbon Week, Peace Builders, and Peace Week. The study authors informed the WWC that these schools had some of the additional programs in place before the time of 
pretest. Because the same practices took place in both study conditions, and because greater time dedicated to those additional practices was reported for the comparison group than for the intervention group, this study 
was reviewed. The WWC cautions that interaction effects between Positive Action and the additional programs may have enhanced the program effects. 

2. The study also examined students’ attitudes and values and additional teacher-reported outcomes in the behavior domain. But, since complete statistical information was not available, these outcomes were not reviewed. 


WWC Intervention Report Positive Action 


October 4, 2006 


6 






Appendix A1.2 Study Characteristics: Flay & Allred, 2003 (quasi-experimental design) 


Characteristic 

Description 

Study citation 

Flay, B. R., & Allred, C. G. (2003). Long-term effects of the Positive Action program. American Journal of Healthy Behavior, 27(1), 6-21. 

Participants 

The study included 36 elementary schools. About 62% of the students in the intervention group participated in the free or reduced-price lunch program compared with 67% 
in the comparison group. About half of the students were white (50.59% in the intervention group and 44.66% in the comparison group). About one-fourth of the students 
were African-American (24.61% in the intervention group and 28.48% in the comparison group). About one-fifth of the students were Hispanic (20.71% in the intervention 
group and 23.23% in the comparison group). Because schools in the intervention and comparison conditions were matched, there were no statistically significant differences 
in demographic characteristics between the two groups. 

Setting 

The participating schools were from one large Southeastern school district. The school district was characterized by a large number of school sites (about 65) that have 
implemented Positive Action for four or more years. 

Intervention 

The program consisted of the Positive Action curriculum and additional activities that involved school principals, parents, and community members. The components 
addressed school and classroom management, learning climate, and skills and knowledge related to core values. 3 

Comparison 

The comparison schools were drawn from the same school district as the intervention schools and were matched on demographic characteristics. Comparison schools did not 
implement the Positive Action program. 

Primary outcomes 
and measurement 4 

The study measures in the behavior domain included violence and suspensions rates. The study measures in the academic achievement domain included the Florida Compre- 
hensive Aptitude Test (FCAT) and grade retention rates. (See Appendices A2.1 and A2.2 for more detailed descriptions of outcome measures.) 

Teacher training 

No information on teacher training was provided. 


3. Half of the schools in the intervention group implemented additional character and behavior programs at the time of the study. The study authors informed the WWC that these schools had the additional programs in place before the time 
of pretest. That is, the Positive Action program was the only new addition between the pretest and the posttest. The WWC cautions that interaction effects between Positive Action and the additional programs may have enhanced the 
program effects. 

4. Flay and Allred (2003) also investigated the long-term impact of Positive Action on students in middle and high schools. The study conducted school-level analyses comparing three categories of schools: less than 60%, between 60% 
and 79%, and between 80% and 100% of the student population with exposure to the elementary school Positive Action program. But, because those analyses did not use a comparison group, they were not reviewed in this WWC report. 
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Appendix A2.1 Outcome measures in the behavior domain 


Outcome measure 

Description 

Violence rates 

School-level archival data that consist of disciplinary referrals for incidents of violence per 100 students (as cited in Flay & Allred, 2003). 

Suspensions rates 

School-level archival data that consist of percent of students who received out-of-school suspensions (as cited in Flay & Allred, 2003). 

Tobacco use 

One item rated on a three-point scale on which students indicate if they have ever smoked cigarettes (as cited in Flay et al., 2006). 

Alcohol use 

One item rated on a three-point scale on which students indicate if they have ever drunk alcoholic beverages (as cited in Flay et al., 2006). 

Being drunk 

One item rated on a three-point scale on which students indicate if they have ever been drunk (as cited in Flay et al., 2006). 

Illegal drug use 

One item rated on a three-point scale on which students indicate if they have ever used illegal drugs (as cited in Flay et al., 2006). 

Violent behavior 

A self-report survey that measures level of engagement in five serious behaviors: carried a knife or razor to use to hurt someone, threatened to cut or stab someone, cut or 
stabbed someone on purpose to hurt them, carried a gun, and shot at someone (as cited in Flay et al., 2006). 

Suspensions 

School-level archival data that consist of the rates at which students were suspended during the school year (as cited in Flay et al., 2006). 


Appendix A2.2 Outcome measures in the academic achievement domain 


Outcome measure 

Description 

The Florida Comprehensive 
Aptitude Test (FCAT) 

This standardized test contains two basic components: criterion-referenced tests (CRT), measuring selected benchmarks in mathematics, reading, science, and writing 
from the Sunshine State Standards (SSS); and norm-referenced tests (NRT) in reading and mathematics, measuring individual student performance against national 
norms. Total score for the grade 4 FCAT was used in the reviewed study (as cited in Flay & Allred, 2003). 

The Hawaii Content and 
Performance Standards 
test (HCPS) 

A state standardized academic achievement test. Fifth-grade test scores on the reading and math subtests were used in this study (as cited in Flay et al., 2006). 

Absentee rates 

School-level archival data that consist of percent of students absent for 21 or more days during the school year (as cited in Flay & Allred, 2003). 

Retention in grade 

School-level archival data that consist of the number of students retained in the same grade level (as cited in Flay et al., 2006). 
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Appendix A3.1 


Summary of study findings included in the rating for the behavior domain 1 





Author’s findings from the study 







Mean outcome 
(standard deviation 2 ) 


WWC calculations 




Sample size 


Mean difference 4 

Statistical 



Study 

(students/ 

Positive Action Comparison 

(Positive Action 

significance 6 

Improvement 

Outcome measure 

sample 3 

schools) 

group group 

-comparison) 

Effect size 5 (at a = 0.05) 

index 7 





Flay et al., 2006 (randomized controlled trial) 





Suspensions 

Grade 5 

2,660/20 

1.63 to 1,000 
students 

6.2 to 1,000 
students 

0.01 

0.81 

Statistically 

significant 

+29 

Tobacco use 

Grade 5 

2,660/20 

3.33 (na) 

4.78 (na) 

1.45 

0.23 

Statistically 

significant 

+9 

Alcohol use 

Grade 5 

2,660/20 

9.63 (na) 

14.83 (na) 

5.20 

0.30 

Statistically 

significant 

+12 

Being drunk 

Grade 5 

2,660/20 

0.74 (na) 

3.35 (na) 

2.61 

0.93 

Statistically 

significant 

+32 

Illegal drug use 

Grade 5 

2,660/20 

0 (na) 

2.39 (na) 

2.39 

1.08 

Statistically 

significant 

+36 

Serious violence (boys) 

Grade 5 

1,330/20 

10.8 (na) 

23.5 (na) 

12.70 

0.56 

Statistically 

significant 

+21 

Serious violence (girls) 

Grade 5 

1,330/20 

5.8 (na) 

3.6 (na) 

-2.20 

-0.30 

ns 

-12 

Average 8 for behavior (Flay et al., 2006) 





0.52 

Statistically 

significant 

+20 




Flay & Allred, 2003 (quasi-experimental design) 




Violence rates 

Grades 1-6 

36 schools 

3.83 (nr) 

12.11 (nr) 

8.28 

0.75 

Statistically 

significant 

+27 

Suspensions rates 

Grades 1-6 

36 schools 

2.72 (nr) 

4.09 (nr) 

1.37 

0.25 

Statistically 

significant 

+10 

Average 8 for behavior (Flay & Allred, 2003) 





0.50 

Statistically 

significant 

+19 

Domain average 8 across studies 






0.51 

na 

+19 


ns = not statistically significant 
nr = not reported 
na = not applicable 

(continued) 
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Appendix A3.1 


Summary of study findings included in the rating for the behavior domain 1 (continued) 


1. This appendix reports findings considered for the effectiveness rating and the improvement index. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. 

3. Flay and Allred (2003) also investigated the impact of Positive Action on secondary schools, but this study did not meet WWC evidence screens because it did not use a comparison group. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. The sign for the direction of effect for outcomes indicating problem behavior was reversed, to indicate a 
positive effect that is associated with a small rate of problem behavior in the intervention schools. 

5. Effect sizes for all outcomes in the behavior domain were calculated using the odds ratio formula. For an explanation of the effect size calculation, please see the Technical Details of WWC-Conducted Computations . 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. The level of statistical significance was reported by the study authors or, where neces- 
sary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch . See the Technical Details of WWC- 
Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Positive Action, corrections for both clustering and multiple comparisons were needed. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values 
between -50 and +50, with positive numbers denoting favorable results. 

8. The WWC-computed average effect sizes are simple averages rounded to two decimal places. The average improvement indices are calculated from the average effect sizes. 
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Appendix A3.2 Summary of study findings included in the rating for the academic achievement domain 1 





Author’s findings from the study 







Mean outcome 
(standard deviation 2 ) 


WWC calculations 




Sample size 


Mean difference 4 

Statistical 



Study 

(students/ 

Positive Action Comparison 

(Positive Action 

significance 6 

Improvement 

Outcome measure 

sample 3 

schools) 

group group 

-comparison) 

Effect size 5 (at a = 0.05) 

index 7 


The Hawaii Content and 
Performance Standards test 
(HCPS) — math (percentage 
reaching proficiency) 

Grade 5 

20/2,660 

Flay et al., 2006 (randomized controlled trial) 

26.00 (na) 21.00 (na) 5.00 

0.17 

ns 

+7 

The Hawaii Content and 
Performance Standards test 
(HCPS) — reading (percentage 
reaching proficiency) 

Grade 5 

20/2,660 

52.00 (na) 

44,00 (na) 

8.00 

0.19 

ns 

+8 

Grade retention 

Grade 5 

20/2,660 

1 to 1,000 students 

6 to 1,000 students 

0.01 

1.09 

Statistically 

significant 

+36 

Average 8 for academic achievement (Flay et al., 2006) 




0.48 

Statistically 

significant 

+19 




Flay & Allred, 2003 (quasi-experimental design) 




The Florida Comprehensive 

Grade 4 

36 schools 

290.90 (nr) 

278.40 (nr) 

12.50 

0.28 

Statistically 

+11 

Aptitude Test (FCAT) 







significant 


Absentee rates 

Grades 1-6 

36 schools 

10.79 (nr) 

12.36 (nr) 

1.57 

0.25 

ns 

+10 

Average 8 for academic achievement (Flay & Allred, 2003) 




0.27 

Statistically 

+11 








significant 


Domain average 8 across studies 





0.38 

na 

+15 


ns = not statistically significant 
nr = not reported 
na = not applicable 


1 . This appendix reports findings considered for the effectiveness rating and the improvement index. One additional finding based on a subtest of the Florida Comprehensive Aptitude Test (FCAT) was not used for rating purposes because 
of an overlap with total FCAT score, but is presented in Appendix A4. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. For the 
calculation of an effect size for the FCAT outcome, the school-level standard deviations reported in the original article were converted to student-level standard deviations. Standard deviations by condition were not reported in the study; 
therefore, pooled standard deviations were used for effect size calculation. 

3. Flay and Allred (2003) also investigated the impact of Positive Action on secondary schools, but this study did not meet WWC evidence screens because it did not use a comparison group. 

(continued) 
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Appendix A3.2 Summary of study findings included in the rating for the academic achievement domain 1 (continued) 


4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. The sign for the direction of effect for absentee rates was reversed, to indicate a positive effect that is 
associated with smaller absentee rates in the intervention schools. 

5. Effect sizes for all outcomes reviewed for the Flay and Allred (2003) study, except for students scores on the FCAT, were calculated using the odds ratio formula. For an explanation of the effect size calculation, please see the Technical 
Details of WWC-Conducted Computations . 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. The level of statistical significance was reported by the study authors or, where neces- 
sary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch . See the Technical Details of WWC- 
Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Positive Action, corrections for both clustering and multiple comparisons were needed. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values 
between -50 and +50, with positive numbers denoting favorable results. 

8. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The domain improvement index is calculated from the average effect size. 
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Appendix A4 Summary of additional study findings for the academic achievement domain 1 





Author’s findings from the study 







Mean outcome 
(standard deviation 2 ) 


WWC calculations 




Sample size 


Mean difference 3 

Statistical 



Study 

(students/ 

Positive Action Comparison 

(Positive Action 

significance 5 

Improvement 

Outcome measure 

sample 

schools) 

group group 

-comparison) 

Effect size 4 (at a = 0.05) 

index 6 





Flay & Allred, 2003 (quasi-experimental design) 




Florida Reading Test 

Grade 4 

36 schools 

105.90 (nr) 73.10 (nr) 32.80 

1.29 

Statistically 

significant 

+40 


nr = not reported 


1 . This appendix presents an additional finding that falls in the academic achievement domain. This is a subtest of the Florida Comprehensive Aptitude Test (FCAT). Findings for total score on the FCAT are presented in Appendix A3. 2. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. For the calcu- 
lation of an effect size pooled standard deviations, which were converted to student-level standard deviations, were used. 

3. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

4. For an explanation of the effect size calculation, please see the Technical Details of WWC-Conducted Computations . 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. The level of statistical significance was reported by the study authors or, where neces- 
sary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch . See the Technical Details of WWC- 
Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Positive Action, a correction for clustering was needed. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values 
between -50 and +50, with positive numbers denoting favorable results. 
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Appendix A5.1 Positive Action rating for the behavior domain 


The WWC rates an intervention’s effects for a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative. 1 

For the outcome domain of behavior, the WWC rated Positive Action as having positive effects. The remaining ratings (potentially positive effects, mixed effects, no 
discernible effects, potentially negative effects, and negative effects) were not considered, because Positive Action was assigned the highest applicable rating. 


Rating received 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1 : Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Met. Positive Action had two studies, one of which met WWC evidence standards for a strong design. Both studies showed statistically significant 
positive effects. 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. The WWC analysis found no statistically significant or substantively important negative effects in this domain. 

1 . For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain level effect. The WWC also considers the size of the domain level effect for ratings of potentially positive effects. See the WWC 
Intervention Rating Scheme for a complete description. 
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Appendix A5.2 Positive Action rating for the academic achievement domain 


The WWC rates an intervention’s effects for a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative. 1 

For the outcome domain of academic achievement, the WWC rated Positive Action as having positive effects. The remaining ratings (potentially positive effects, 
mixed effects, no discernible effects, potentially negative effects, and negative effects) were not considered, because Positive Action was assigned the highest appli- 
cable rating. 

Rating received 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1 : Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Met. Positive Action had two studies, one of which met WWC evidence standards for a strong design. Both studies showed statistically significant 
positive effects. 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. The WWC analysis found no statistically significant or substantively important negative effects in this domain. 

1 . For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain level effect. The WWC also considers the size of the domain level effect for ratings of potentially positive effects. See the WWC 
Intervention Rating Scheme for a complete description. 
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